Remote operating system

ABSTRACT

Provided is a remote operating system which enables to operate a device by a remote operating device in a simple and reliable manner even if the device to be operated is not equipped with a communication function, or if displayed contents on a monitor screen of the device to be operated are not recognized by the remote operating device. A remote operating system ( 100 ) includes: a screen acquisition unit ( 121 ); an image determination unit ( 122 ); a state providing unit ( 123 ); an operation reception unit ( 131 ); an operating signal output unit ( 132 ); a correct or incorrect operation determination unit ( 133 ); and a state database DB. The remote operating system ( 100 ) further includes a capture device ( 141 ) for capturing an image of a monitor X 1  of a device X of interest, and a control interface ( 142 ) connected to a keyboard X 2  of the object X to be controlled in a superimposed manner.

TECHNICAL FIELD

The present invention relates to a remote operating system remotelyoperating a device to be operated by a remote operating device based ona state of a monitor screen of the device to be operated.

BACKGROUND ART

Conventionally there is known, as a remote operating system of thiskind, a system in which a device to be operated is configured to enablecommunication with a remote operating device, and in which an operationscreen of the device to be operated is displayed on the remote operatingdevice side to allow a remote operation to be received, as described inPatent Literature 1 shown below.

CITATION LIST Patent Literature

Patent Literature 1: JP 2013-62778 A

SUMMARY OF INVENTION Technical Problem

However, in such a conventional remote operating system, in a case wherethe device to be operated is not equipped with a communication function,or in which the monitor screen of the device to be operated merelydisplays contents (for example, display of an analog screen) and is notrecognized on the remote operating device side, there is a problem thatit is difficult to operate the device to be operated by the remoteoperating device.

In consideration of the above problem, an object of the presentinvention is to provide a remote operating system enabling a device tobe operated to be operated by a remote operating device in a simple andreliable manner even in a case where the device to be operated is notequipped with a communication function, or in which displayed contentson a monitor screen of the device to be operated are not recognized onthe remote operating device side.

Solution to Problem

According to a first aspect of the present invention, there is provideda remote operating system remotely operating a device to be operated bya remote operating device based on a state of a monitor screen of thedevice to be operated, including:

a screen acquisition unit configured to capture the monitor screen ofthe device to be operated via a capture device and acquiring the monitorscreen as an image;

an image determination unit configured to determine a state of the imagefrom the image acquired by the screen acquisition unit;

a state providing unit configured to provide the remote operating devicewith the state determined by the image determination unit;

an operation reception unit configured to receive an operation from theremote operating device; and

an operating signal output unit configured to convert the operationreceived by the operation reception unit into an operating signal forthe device to be operated and outputting the operating signal to thedevice to be operated via a control interface,

wherein the image determination unit determines whether the state of theimage is in an image state that is operable by the device to be operatedfrom the image acquired by the screen acquisition unit, and in a casewhere the state of the image is the image state that is operable by thedevice to be operated, the state providing unit generates an operationinput screen corresponding to the operable image state and provides theoperation input screen to the remote operating device.

With the remote operating system according to the first aspect of thepresent invention, the screen acquisition unit captures the monitorscreen of the device to be operated and acquires the monitor screen asthe image. Accordingly, even in a case where the monitor screen of thedevice to be operated merely displays an analog screen, information onthe monitor screen can be provided to the remote operating device.

While various images are displayed on the monitor screen, the imagedetermination unit determines the state of the capture image acquired bythe screen acquisition unit in advance, and the state of the image isprovided to the remote operating device. Accordingly, screen informationappropriate for remote operation can be provided to the remote operatingdevice.

Also, with the remote operating system according to the first aspect ofthe present invention, the operation in the remote operating devicereceived by the operation reception unit is converted into the operatingsignal for the device to be operated and is output to the device to beoperated. Accordingly, even in a case where the device to be operated isnot equipped with a communication function, the device to be operatedcan be operated directly by the operating signal.

In this manner, with the remote operating system according to the firstaspect of the present invention, even in a case where the device to beoperated is not equipped with a communication function, or in whichdisplayed contents on the monitor screen of the device to be operatedare not recognized on the remote operating device side, the device to beoperated can be operated by the remote operating device in a simple andreliable manner.

Meanwhile, in the remote operating system according to the first aspectof the present invention, the capture of the monitor screen includes notonly capture by hardware using the capture device but also capture bysoftware using a remote desktop.

Also, with the remote operating system according to the first aspect ofthe present invention, the image determination unit determines whetherthe state of the image acquired from the screen acquisition unit is animage state that is operable by the device to be operated. In a casewhere the state of the image is the image state that is operable by thedevice to be operated, the state providing unit generates an operationinput screen corresponding to the operable image state and provides thescreen to the remote operating device. Accordingly, the operation inputscreen appropriate for remote operation can be provided to the remoteoperating device.

In this manner, with the remote operating system according to the firstaspect of the present invention, even in a case where the device to beoperated is not equipped with a communication function, or in whichdisplayed contents on the monitor screen of the device to be operatedare not recognized on the remote operating device side, the device to beoperated can be operated by the remote operating device in a simple andreliable manner through the operation input screen appropriate forremote operation.

As the remote operating system according to a second aspect of thepresent invention in the remote operating system according to the firstaspect of the present invention, the operating signal output unit isconnected to a human interface (for example, a keyboard) of the deviceto be operated in a superimposed manner, converts the operation receivedby the operation reception unit into the operating signal for the deviceto be operated as an emulation signal for the human interface, andoutputs the operating signal to the device to be operated.

With the remote operating system according to the second aspect of thepresent invention, the operating signal output unit converts theoperation received by the operation reception unit into the operatingsignal for the device to be operated as the emulation signal for thehuman interface of the device to be operated and outputs the operatingsignal to the device to be operated.

Thus, with the remote operating system according to the second aspect ofthe present invention, even in a case where the device to be operated isnot equipped with a communication function or does not have an externalconnection input, the device to be operated can be operated by theremote operating device in a simple and reliable manner.

The remote operating system according to a third aspect of the presentinvention in the remote operating system according to the first orsecond aspect of the present invention includes a correct or incorrectoperation determination unit configured to determine whether theoperation is correct or incorrect based on the state of the imageacquired via the screen acquisition unit after the operating signaloutput unit outputs the operating signal to the device to be operated.

With the remote operating system according to the third aspect of thepresent invention, the correct or incorrect operation determination unitdetermines whether the operation is correct or incorrect based on thestate of the image acquired via the screen acquisition unit after theoperating signal output unit outputs the operating signal to the deviceto be operated. Thus, it is possible to confirm based on a change of thestate of the device to be operated if the remote operation has beenexecuted reliably.

In this manner, with the remote operating system according to the thirdaspect of the present invention, even in a case where the device to beoperated is not equipped with a communication function, or in which adisplay content on the monitor screen of the device to be operated isnot recognized on the side of the remote operating device, the device tobe operated can be operated by the remote operating device in a simpleand reliable manner, and execution of the operation can be confirmed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a system configuration diagram illustrating an overview of aremote operating system according to an embodiment.

FIG. 2 is a flowchart illustrating a processing content in the remoteoperating system in FIG. 1.

FIG. 3 is a flowchart illustrating a processing content in the remoteoperating system in FIG. 1.

DESCRIPTION OF EMBODIMENTS

As illustrated in FIG. 1, a remote operating system 100 according to anembodiment is a remote operating system in which, even in a case where adevice X to be operated serving as a device to be operated is notequipped with a communication function or does not have an externalconnection interface, or in which a screen displayed on a monitor X1 ofthe device X to be operated is an analog screen, the device X to beoperated is remotely operated by a remote operating device Y based on astate of the screen on the monitor X1 of the device X to be operated.

The remote operating system 100 includes a screen acquisition unit 121,an image determination unit 122, a state providing unit 123, anoperation reception unit 131, an operating signal output unit 132, acorrect or incorrect operation determination unit 133, and a statedatabase DB.

The remote operating system 100 also includes a capture device 141capturing an image of the monitor X1 of the device X to be operated anda control interface 142 connected to a keyboard X2 serving as a humaninterface of the object X to be controlled in a superimposed manner.

Meanwhile, the screen of the monitor X1 of the device X to be operatedis a screen of a computer, for example, at a front desk of anaccommodation facility, serving as the device X to be operated. On thescreen, for example, “Occupied,” “Unoccupied,” “Cleaning,” “Reserved,”and the like are displayed and color-coded at regions divided based onthe rooms.

The screen acquisition unit 121 captures and acquires an image of thescreen on the monitor X1 via the capture device 141. The capture device141 outputs to the remote operating system 100 image data acquired bysampling an analog screen signal of a monitor cable (VGA cable)connecting the front computer X to the monitor X1 at a constantfrequency.

For example, the capture device 141 may be a VGA video capture board ormay have a function of a VGA distribution cable as well as a function ofthe video capture board. Also, the function of the capture device 141 isnot limited to video capturing from the VGA cable, and the capturedevice 141 may be a video capture board capturing a video through avideo (an S-Video type or a composite type) input/output cable, a DVIcapture board for digital display, an SDI capture board for a video forbusiness purposes or for a security camera, an HDMI (registeredtrademark) capture board for general video devices, or the like.

The image determination unit 122 determines a state of the image fromthe image acquired from the screen acquisition unit 121. Meanwhile, theimage determination unit 122 determines the state of the image acquiredfrom the screen acquisition unit 121 after performing filter processingto the image as needed. The filter processing is RGBA conversion, forexample, in which a capture image is converted into an image having aspecified color number. It is to be noted that the filter processing maybe PNG conversion or the like having a smaller color number instead ofthe RGBA conversion.

Specifically, the image determination unit 122 determines whether thestate of the image acquired from the screen acquisition unit 121 is ascreen state that is operable by the device X to be operated.

The state providing unit 123 provides the remote operating device Y withthe state of the screen of the device X to be operated determined by theimage determination unit.

Specifically, in a case where, as a result of the determination by theimage determination unit 122, the state of the image of the device X tobe operated is an image state that is operable by the device X to beoperated, the state providing unit 123 generates an operation inputscreen corresponding to the operable image state and provides the screento the remote operating device Y.

The components 121 to 123 in the remote operating system 100, whichdetermine a state of an image on the monitor X1 of the device X to beoperated and provide the state to the external remote operating deviceY, have been described above.

The operation reception unit 131 receives an operation from the remoteoperating device Y in the remote operating system 100. It is to be notedthat the remote operating device Y may be a personal computer Y1 or asmartphone Y2 as long as the remote operating device Y is an externaloperating device connected via a network NW such as the Internet.

The operating signal output unit 132 converts the operation in theremote operating device Y received by the operation reception unit 131into an operating signal for the device X to be operated and outputs thesignal to the device X to be operated.

More specifically, the operating signal output unit 132 outputs thesignal to the control interface 142 so that the operation received bythe operation reception unit 131 becomes an emulation signal for thekeyboard X2 of the device X to be operated via the control interface 142connected to the keyboard X2 of the device X to be operated in asuperimposed manner.

After the operating signal output unit 132 outputs the operating signalto the device X to be operated, the correct or incorrect operationdetermination unit 133 determines whether the operation is correct orincorrect based on a state of an image acquired via the screenacquisition unit 121.

The components 131 to 133 in the remote operating system 100, whichoperate the device X to be operated by the operation from the remoteoperating device Y, have been described above.

Also, in the remote operating system 100, the state of the imagedetermined by the image determination unit 122 and the correct orincorrect operation determined by the correct or incorrect operationdetermination unit 133 are stored in the state database DB.

The configuration of the remote operating system 100 has been describedabove. For example, the remote operating system 100 is constituted byhardware such as a CPU (Central Processing Unit), a ROM (Read OnlyMemory), and a RAM (Random Access Memory), stores programs that executeprocessing by the above processing means 121 to 123, 131 to 133, and DBin a memory (not illustrated), and executes the programs to function asa computing device (sequencer) configured to execute the above controlprocessing.

Next, referring to FIG. 2, described is a processing for determining astate of an image on the monitor X1 of the device X to be operated andproviding the state to the external remote operating device Y in theremote operating system 100.

First, the remote operating system 100 acquires an image obtained bycapturing a screen on the monitor X1 (actually, an image captured at aconstant frequency) via the capture device 141 by the screen acquisitionunit 121 (STEP 121 in FIG. 2).

Subsequently, the remote operating system 100 determines whether a stateof the capture image acquired in STEP 121 is a screen state that isoperable by the device X to be operated by the image determination unit122 (STEP 122 in FIG. 2).

As a specific determination method, various methods can be employed. Forexample, the remote operating system 100 acquires color information at aspecified point on the image and determines a state of the image on thescreen of the monitor X1 of the device X to be operated (whether thestate is an image state that is operable by the device X to be operated,and in a case where the state is the operable image state, how the imagestate is) based on combination of colors of the acquired colorinformation (RGB data).

Also, as another determination method, for example, a state of the imageon the screen of the monitor X1 of the device X to be operated may bedetermined by recognizing characters (including character strings)displayed on the image by OCR (optical character recognition).

Further, to improve determination accuracy, a state of the image on thescreen of the monitor X1 of the device X to be operated may bedetermined based on combination of the color information on the imageand the characters recognized by the OCR (optical characterrecognition).

Hence, for example, in a case where the device X to be operated is afront computer in an accommodation facility, at the time of capturingthe image on the monitor X1, the remote operating system 100 determinesthat the image on which “Occupied,” “Unoccupied,” “Cleaning,”“Reserved,” and the like are displayed and color-coded at regionsdivided based on the rooms is in an operable state and is in a statewhere a reservation for an unoccupied room is receivable.

The remote operating system 100 then stores a result determined by theimage determination unit 122 in the state database DB (STEP 123 in FIG.2).

Subsequently, the state providing unit 123 extracts an operable functionbased on the state of the image on the screen of the monitor X1 of thedevice X to be operated determined by the image determination unit 122(determination result read from the state database DB) (STEP 124 in FIG.2).

Specifically, the state providing unit 123 refers to a state table inwhich determination results and operable functions are listed in amatrix form to extract an operable function corresponding to thedetermination result by the image determination unit 122.

Subsequently, the state providing unit 123 generates an operation inputscreen (STEP 125 in FIG. 2).

Specifically, the state providing unit 123 generates the operation inputscreen corresponding to the operable function extracted in STEP 124.

Here, the operation input screen is a screen that can be displayed byoutput to a browser in HTML, or with use of a dedicated application. Forexample, in a case where the device X to be operated is a front computerin an accommodation facility, an operation input screen in whichreservation buttons are displayed only at unoccupied room numbers isgenerated.

The operation input screen is generated not as a screen simply listingthe operable functions but as a screen providing only necessaryfunctions in accordance with additional conditions.

Examples of the additional conditions are a condition in which, when thenumber of unoccupied rooms is three or less, reservation buttons servingas reservation receiving functions are not displayed to stop receivingreservations, and a condition in which, during a specified period oftime (2 to 5 a.m.), reservation buttons serving as reservation receivingfunctions are not displayed to stop receiving reservations during theperiod.

Subsequently, the state providing unit 123 provides the operation inputscreen generated in STEP 125 via the network NW to the remote operatingdevice Y (STEP 126 in FIG. 2).

The above describes the processing for determining a state of an imageon the monitor X1 of the device X to be operated and providing the stateto the external remote operating device Y in the remote operating system100. According to the processing, even in a case where the screen of themonitor X1 of the device X to be operated merely displays an analogscreen, information on the screen of the monitor X1 can be provided tothe remote operating device.

Also, while various images are displayed on the screen of the monitorX1, the image determination unit 122 determines in advance, a state of acapture image acquired by the screen acquisition unit 121, and the stateproviding unit 123 generates an operation input screen corresponding tothe state of the image and provides the operation input screen to theremote operating device Y. Accordingly, screen information appropriatefor remote operation can be provided to the remote operating device Y.

Meanwhile, as for the timing to generate the operation input screen bythe state providing unit 123, the operation input screen may begenerated at regular intervals, or a change of an image state may bemonitored at regular processing cycles, and the operation input screenmay be generated at a timing when the image state changes.

Also, the timing to generate the operation input screen by the stateproviding unit 123 may be a timing when the remote operating device Yaccesses the remote operating system 100 or a timing when the remoteoperating device Y requests operation of the device X to be operated. Inthis case, the state providing unit 123 preferably provides thegenerated operation input screen only to the remote operating device Ythat has accessed the remote operating system 100 or the remoteoperating device Y that has requested the operation.

Next, referring to FIG. 3, described is a processing for operating thedevice X to be operated by the operation in the remote operating deviceY in the remote operating system 100.

As a prerequisite, as described above, when the state of the image onthe monitor X1 of the device X to be operated is determined, and theoperation input screen corresponding to the content is provided to theexternal remote operating device Y, a user of the remote operatingdevice Y decides the operation content for the device X to be operatedwhile he/she is looking at the operation input screen.

At this time, the remote operating system 100 monitors whether or notthere is an operation request from the remote operating device Y atregular processing cycles by the operation reception unit 131 (STEP 131in FIG. 3).

In a case where there is an operation request from the remote operatingdevice Y (YES in STEP 131 in FIG. 3), the operation reception unit 131specifies the operation content (STEP 132 in FIG. 3). Conversely, in acase where there is no operation request from the remote operatingdevice Y (NO in STEP 131 in FIG. 3), the operation reception unit 131repetitively executes the monitoring until there is an operationrequest.

Specifying the operation content is performed by specifying theoperation content on the operation input screen. More specifically,specifying the operation content is performed by use of a URL parametertransmitted from the remote operating device Y to the remote operatingsystem 100.

For example, in a case where the device X to be operated is a frontcomputer in an accommodation facility, and in which a reservationoperation for room 101, which is unoccupied, is performed in the remoteoperating device Y, the operation of the reservation button for room 101is specified by use of a URL parameter of [http://server address/yoyaku(operation command)?room (targeted unoccupied room)=101].

Subsequently, the operating signal output unit 132 converts theoperation content specified in STEP 132 into an operating signal for thedevice X to be operated (STEP 133 in FIG. 3).

Specifically, converting into the operating signal is executed byconverting the operation content into a control signal (executioncommand) for the control interface 142 corresponding to the operationcontent so that the operation received by the operation reception unit131 may be an emulation signal for the keyboard X2 of the device X to beoperated via the control interface 142 connected to the keyboard X2 ofthe device X to be operated in a superimposed manner.

Here, a control signal (execution command) causing a delay (delay time)that is likely to be generated when a person operates the keyboard X2 isinserted as needed into the control signal (execution command) to thecontrol interface 142.

Subsequently, the operating signal output unit transmits the operatingsignal for the device X to be operated converted in STEP 133 to thecontrol interface (STEP 134 in FIG. 3).

The control interface 142 thus converts the control signal (executioncommand) into a control signal (operation code) that the device X to beoperated can recognize and outputs the control signal (operation code)to the device X to be operated. Similarly, a control signal (executioncommand) causing a delay (delay time) that is likely to be generatedwhen a person operates the keyboard X2 is inserted as needed into thecontrol signal (operation code).

For example, in a case where the device X to be operated is a frontcomputer in an accommodation facility, and in which a reservationoperation for room 101, which is unoccupied, is performed in the remoteoperating device Y, a control signal (operation code) corresponding to akey operation for room number 101 on the keyboard X2, that is, areservation key operation, is generated in the form of containing acontrol signal (execution command) causing a delay (delay time) and isoutput to the device X to be operated.

Meanwhile, in the present embodiment, although a case where the controlsignal (operation code) that the device X to be operated can recognizeis generated via the control interface is described, the convertedcontrol signal (operation code) may be output directly by the operatingsignal output unit 132 of the remote operating system 100.

The above describe the processing for operating the device X to beoperated by operation in the remote operating device Y in the remoteoperating system 100. In addition to the processing, correct orincorrect operation determination processing is preferably performed bythe correct or incorrect operation determination unit 133 (STEPs 135 to136 in FIG. 3).

Specifically, after the operating signal is output to the device X to beoperated by the operating signal output unit 132 (actually, after apredetermined period of time has passed based on a counter timer or thelike), the correct or incorrect operation determination unit 133determines whether the operation is successful based on a state of animage acquired via the screen acquisition unit 121 (STEP 135 in FIG. 3).

Here, in the analysis of the state of the image acquired via the screenacquisition unit 121, in a similar manner to the processing of the imagedetermination unit 122 (refer to STEP 122 in FIG. 2), in a case wherethere is a change of the state of the image that is operable by thedevice X to be operated corresponding to the operation content, thecorrect or incorrect operation determination unit 133 determines thatthe operation is successful (YES in STEP 135 in FIG. 3) and ends thesequential processing.

Conversely, in a case where there is no change of the state of the imagethat is operable by the device X to be operated corresponding to theoperation content, the correct or incorrect operation determination unit133 determines that the operation is not successful (NO in STEP 135 inFIG. 3), notifies the remote operating device Y of an error (STEP 136 inFIG. 3), and ends the sequential processing.

Meanwhile, instead of the error notification, (the processing may returnto STEP 134, and) the operating signal output unit 132 may transmit theoperating signal again (refer to STEP 134).

Also, the determination result (image state) by the correct or incorrectoperation determination unit 133 is preferably (overwritten and) storedin the state database DB.

The above describe the processing for operating the device X to beoperated by the operation by the remote operating device Y in the remoteoperating system 100 and processing for determining whether theoperation is correct or incorrect.

In the processing, the operation in the remote operating device Yreceived by the operation reception unit 131 is converted into theoperating signal for the device X to be operated and is output to thedevice X to be operated. Accordingly, even in a case where the device Xto be operated is not equipped with a communication function, the deviceX to be operated can be operated directly by the operating signal.

Also, since the correct or incorrect operation determination unit 133performs the correct or incorrect operation determination based on achange of a state of the device X to be operated, it is possible toconfirm if the remote operation has been executed reliably.

In this manner, with the remote operating system according to thepresent embodiment, even in a case where the device X to be operated isnot equipped with a communication function, or in which a displaycontent on the screen of the monitor X1 of the device X to be operatedis not recognized on the side of the remote operating device Y, thedevice X to be operated can be operated by the remote operating device Yin a simple and reliable manner, and execution of the operation can beconfirmed.

Meanwhile, in the present embodiment, a case where capture of themonitor screen is capture by hardware using the capture device 141 hasbeen described. However, the capture is not limited to this and may becapture by software using a remote desktop.

Also, in the present embodiment, a case where the keyboard X2 isemulated as a human interface of the object X to be controlled has beendescribed. However, the human interface is not limited to this and maybe any human interface such as a mouse, a trackball, and a mechanicalpush button as long as the human interface enables a person to performinput and can be emulated.

Further, in the present embodiment, a case where a personal computer isoperated as the device X to be operated has been described. However, thedevice X to be operated is not limited to this and may be a printerdevice X3 or the like connected to the personal computer.

REFERENCE SIGNS LIST

100: remote operating system, 121: screen acquisition unit, 122: imagedetermination unit, 123: state providing unit, 131: operation receptionunit, 132: operating signal output unit, 133: correct or incorrectoperation determination unit, 141: capture device, 142: controlinterface, X: device to be operated, X1: monitor, X2: keyboard (humaninterface), X3: printer, Y: remote operating device, DB: state database,NW: network

1. A remote operating system remotely operating a device to be operatedby a remote operating device based on a state of a monitor screen of thedevice to be operated, the remote operating system comprising: acomputing device; a capture device configured to capture the monitorscreen of the device to be operated; and a control interface connectedbetween the remote operating system and a human interface of the deviceto be operated in a superimposed manner, wherein the remote operatingsystem, the device to be operated, and the remote operating device areall separate from each other and, among the remote operating system, thedevice to be operated, and the remote operating device, only the remoteoperating system and the remote operating device are in communicationwith each other via a network, wherein the computing device comprises aprocessor and memory storing programs that, when executed by theprocessor, configure the processor to perform the functions comprising:acquiring the monitor screen captured by the capture device as an image;determining a state of the image from the image acquired by the capturedevice; providing the remote operating device with the state determined;receiving an operation communicated to the computing device from theremote operating device; and converting the operation received from theremote operating device into an operating signal for the device to beoperated as an emulation signal for the human interface, and outputtingthe operating signal to the device to be operated via the controlinterface, wherein the determining further comprises determining fromthe image acquired whether the state of the image is in an image statethat is operable by the device to be operated based on the operationcommunicated from the remote operating device, in a case where the stateof the image is the image state that is operable by the device to beoperated, the providing further generates an operation input screencorresponding to the operable image state and provides the operationinput screen to the remote operating device, and wherein the determiningfurther determines whether the state of the image is in the image statethat is operable by the device to be operated from the image acquired byrecognizing characters displayed on the image acquired and determining astate of the characters.
 2. (canceled)
 3. The remote operating systemaccording to claim 1, wherein the processor of the computing device isfurther configured to perform the function of: determining whetheroperation of the device to be operated is correct or incorrect based onthe state of the image acquired after outputting the operating signal tothe device to be operated.
 4. A remote operating system remotelyoperating a device to be operated by a remote operating device based ona state of a monitor screen of the device to be operated, the remoteoperating system comprising: a computing device; a capture deviceconfigured to capture the monitor screen of the device to be operated;and a control interface connected between the remote operating systemand a human interface of the device to be operated in a superimposedmanner, wherein the remote operating system, the device to be operated,and the remote operating device are all separate from each other and,among the remote operating system, the device to be operated, and theremote operating device, only the remote operating system and the remoteoperating device are in communication with each other via a network,wherein the computing device comprises a processor and memory storingprograms that when executed by the processor, configure the processor toperform the functions comprising: acquiring the monitor screen capturedby the capture device as an image; recognizing characters displayed onthe image acquired; determining a state of the characters. determiningwhether a state of the image acquired by the capture device is in animage state that is operable by the device to be operated based on thestate of the characters determined; generating an operation input screencorresponding to the operable image state and providing the operationinput screen to the remote operating device, in a case where the stateof the image is the image state that is operable by the device to beoperated; receiving an operation communicated to the computing devicefrom the remote operating device; and converting the operation receivedfrom the remote operating device into an operating signal for the deviceto be operated as an emulation signal for the human interface, andoutputting the operating signal to the device to be operated via thecontrol interface.
 5. The remote operating system according to claim 4,wherein the processor of the computing device is further configured toperform the function of: determining whether operation of the device tobe operated is correct or incorrect based on the state of the imageacquired after outputting the operating signal to the device to beoperated.